This version of the Tokenizer is using a public domain version of Forth (called ThisForth), that was developed by Wil Baden and modified by Apple. The SIOW version allows you to tokenize your Open Firmware FCode without requiring MPW or any other license fees.
Simply launch the Tokenizer by double-clicking. It will create an SIOW window, display some version information, then prompt for input.
To tokenize a file, type:
tokenize <filename>
This will start tokenizing the top-level file. Within this file, a tokenizer FLOAD command can be used to include other files. This command looks like:
fload <filename>
FLOADs can be nested.
The tokenizer builds the FCode image into an internal buffer. After all the source has been tokenized, the image is saved by using the save-image command as:
save-image <filename>
This generates a binary image of the ROM image, which is padded to a multiple of 512 bytes, as required by the PCI standard.
Other formats can be output by the tokenizer:
dump-83 <filename>
This format is a common one for use by PROM programmers.
dump-85 <filename>
This format is an format utilizing only ASCII characters, similar to uuencode. The format is described in detail in "Portable Document Format Reference Manual", by Adobe, where it is called ASCII base-85 encoding.
Use the Quit menu to exit the program.
In addition to the standard Open Firmware words, several additional words have
been added to make it easier to produce PCI Expansion ROM images. These are:
fcode-version2
fcode-end
pci-header
pci-end
encode-file
encode-file-reg
fcode-date
fcode-time
FCODE-VERSION2
A command that generates an FCode header using START1. This is meant to be paired with FCODE-END.
FCODE-END
A command that generates an END0 FCode and completes the information in the FCode header that was generated by FCODE-VERSION2.
PCI-HEADER
A macro that generates a PCI header. PCI-HEADER expects 3 values on the top of the stack when it is invoked: Vendor#, Device# and Class-Code. These values are used for the corresponding fields in the PCI Data structure that is built by the PCI-HEADER. The easiest way of including these values is to put them on the stack outside of the tokenization by using TOKENIZER[ and ]TOKENIZER.
PCI-END
A macro that completes the PCI header created by PCI-HEADER. It fills out the image to a multiple of 512 bytes, and sets the image-length field of the PCI header.
This would generate a PCI Data structure with the Vendor# == 0x1234, Device# == 0x5678 and Class-Code == 0xABCDEF.
NOTE: It is VERY important that your ROM image contain the proper values for Vendor#, Device# and Class-Code. These values are checked during the process of locating an appropriate FCode image for the device; if you do not have values matching your device, your FCode will not be recognized!!!!
ENCODE-FILE
ENCODE-FILE is used for embedding device drivers, etc. as properties. It will build a prop-encoded-array, similar to that produced by ENCODE-BYTES, except that the data will be taken from the specified file during tokenization.
ENCODE-FILE is followed by the filename to be included; thus, its syntax is:
ENCODE-FILE <filename>
Note that since the filename is obtained using Forth's WORD, it must not contain any embedded blanks. The following example shows how a property would be generated by means of ENCODE-FILE.
encode-file xyz.bin " xyz-prop" property
The property value of " xyz-prop" would be the contents of the original file, byte-for-byte; the prop-len of the created property is the same as the length of the original file.
ENCODE-FILE-REG
ENCODE-FILE-REG is used to embed device drivers, etc. within the FCode ROM image, but without consuming device-tree space. The data from the specified file is copied into the ROM image after all of the FCode for has been generated; i.e., at PCI-END time.
ENCODE-FILE-REG is followd by the filename to be included; its syntax is:
ENCODE-FILE-REG <filename>
The property value produced is that of a "reg" property for PCI; i.e., it consists of 5 integers (encoded as with ENCODE-INT) that represent the offset within this FCode image of the file's data. The "reg"'s phys.hi value (i.e., the config-address) will reference the Expansion ROM; it will always have a h# 30 as the low-order byte. The phys.lo value will be the offset of the data (within the Expansion ROM) and the size.lo will be the size of the data.
Note: the code generated by ENCODE-FILE-REG relies upon the value of the "fcode-rom-offset" property that should be generated by Open Firmware before starting the evaluation of the FCode.
Note that to get to the data, you have to add in the value of the "assigned-addresses" property for the Expansion ROM. I.e., the value of the encoded property is relative to the base address of the Expansion ROM. The actual address assigned by Open Firmware to the Expansion ROM is not known at the time that the FCode is evaluated.
Note: due to a bug in the Expansion Manager code, DO NOT USE the ENCODE-FILE-REG form for MacOS drivers. However, other drivers should be encoded with this mechanism to reduce device-tree space.
FCODE-DATE
FCODE-DATE creates an FCode string constant that contains the date of the tokenization (in MM/DD.YY format). This string could be used to create a device tree property that can be inspected by run-time drivers, etc. to check when the image was created.
FCODE-TIME
FCODE-TIME creates an FCode string constant that contains the time of the tokenization (in HH:MM:SS format). This string could be used to create a device tree property that can be inspected by run-time drivers, etc. to check when the image was created.
